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(54) Method of speeding the execution of neural networks for correlated signal processing 



(57) The method allows speeding up the execution 
of a wide class of neural networks fbr processing input 
signals evolving slowly through time, such as, for 
instance, voice, radar, sonar, video signals, and it 
requires no specialised, costly or hard-to-find hardware. 
The method requires storing, fbr the neurons in at least 
one level of the network, the activation value at a certain 
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instant and comparing it with the one computed at the 
subsequent instant. If the activation is equal, the neuron 
carries out no activity, othenmse it propagates the differ- 
ence in activation, multiplied by the interconnection 
weights, to the neurons it is connected to. 
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Description 

The present invention relates to automatic signal 
recognition systems and in particular it concerns a 
method of speeding up the execution of neural networks 
to process correlated signals. 

A neural network is a parallel processing model 
reproducing, in extremely simplified form, the cerebral 
cortex organization. A neural network comprises multi- 
ple processing units, known as neurons, strongly inter- 
connected by means of links of various intensities, 
called synapses or interconnection weights. 

Neurons in general are arranged along a multi-level 
structure, with an input level, one or more intermediate 
levels and an output level. Starting from the input units, 
which receive the signal to be processed, processing 
propagates to the subsequent levels of the network up 
to the output units, which provide the result. A neural 
network is not programmed, but it is trained by means of 
a series of examples of the phenomenon to be mod- 
elled. Various enlbodiments of neural networks are 
desCTibed. for instance, in the book by D. Rumelhart 
"Parallel Distributed Processing", Vol. 1 Foundations. 
MIT Press, Cambridge. N/1ass., 1986. 

Neural network technology can be applied to many 
sectors, such as function estimation, audio and video 
signal processing and classification, automatic controls, 
forecasting and optimisation, although it still presents 
some problems stemming from the lack of means that 
are powerful enough in terms of processing power and 
speed. It is well known that the execution of a neural 
network, when it is carried out by emulation on a 
sequential processor, is very txjrdensome, especially in 
cases requiring networks with many thousands of 
weights, ff the need arises to process, in real time, sig- 
nals continuously varying through time, such as for 
example voice, video, sonar or radar signals, use of this 
technology takes on additional difficulties. 

The known solutions to this problem have taken 
three routes. 

The f irst is aimed at deaeasing the network size by 
pruning weights and units, as described for example by 
R. Reed in "Pruning Algorithms - A Sun^ey" in IEEE 
Transactions on Neural Networks, Vol. 4, no. 5, 1993. 
These methods, however, have limited effectiveness 
since the number of weights and units that can be 
pruned without incurring a degradation in performance 
is often very limited. 

A second route is based on implementation of the 
neural network on a VLSI chip, by exploiting its intrinsic 
potential for paranelisation. This method is. potentially, 
very promising, but it is not very mature yet. It also 
entails the use of specialised hardware, which is often 
very expensive and not easy to Integrate with commer- 
cial processors. 

A third route is the use of specialised hardware of 
multi-processor type, by distributing the execution of the 
neural network among various processors. As in the 
previous case, this possible solution also requires non- 



standard hardware, which is costly and difficult to inte- 
grate with commercial platforms like personal comput- 
ers or workstations. 

The aforesaid drawbacks are obviated by ttie 

5 mettiod of speeding up tiie execution of neural network 
for con'elated signal processing, according to tiie 
present invention, which allows speeding up the execu- 
tion of a wide class of neural networks to process 
sequential input signals evolving slowly through time. 

10 such as, for instance, voice, radar, sonar, video signals, 
and which requires no specialised, costly or hard to find 
hardware. 

In particular, the object of ti^e present Invention is to 
provide a method of speeding up the execution of neural 
15 networks for correlated signal processing, as defined in 
tiie characterising part of claim 1 . 

These and other characteristics of tiie present 
invention shall be clarified better by tiie following 
description of a preferred embodiment of tiie same, 
20 given only by way of non limiting example, and by the 
enck}sed drawings in which: 

Figure 1 depicts a Multi-layer Perceptron neural 
network: 

25 Rgure 2 depicts an individual neuron i with its for- 
ward connections; 

Figure 3 depicts tiie quantization of the co-domain 
of the sigmoid transfer function of the neuron. 

30 The idea tiie method is based upon is tiie following: 
since tiie input signal is sequential and evolves slowly 
and continuously through time, it is not necessary to 
compute again all the activation values of all neurons for 
each input, but rather it is enough to propagate through 

35 tiie network the differences witti respect to tiie previous 
input. That is, tiie operation does not consider ttie abso- 
lute neuron activation values at time t. but the differ- 
ences with respect to activation values at time t-1. 
Therefore at any point of ttie network if a neuron has at 

40 time t an activation that is sufficientiy similar (preferably 
identical) to ttiat of time t-1. then the neuron will not 
propagate any signal fbnward. Since the execution of the 
network is based on the elementary operation of propa- 
gating ttie activation value of each neuron, averaged by 

45 ttie synaptic weights, towards the subsequent neurons, 
it is readily understood ttiat limiting such activity to just 
the neurons for which there is an appreciat)le change in 
activation level is quite important. 

By exploiting this idea, the execution of ttie neural 

50 network can be speeded up by propagating signrticant 
activation differences, and this allows saving up to two 
ttiirds of execution time in case of speech recognition. 
This method requires a very small amount of auxiliary 
memory and it does not entail an appreciable degrada- 

65 tion in performance, as it was experimentally verified. 
Rgure 1 shows a Multi-layer Perception neural net- 
work like the one described in ttie already mentioned 
book by D. Rumelhart "Parallel Distributed Processing". 
The network input is a signal sampled in time and ttie 
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network output are values corresponding to the desired 
processing, for instance input signal classification. 

Each neuron of the intermediate and output levels 
computes its total Input net; as a summation of the prod- 
ucts of the activations Oj of the lower level neurons by s 
the interconnection weights Wjj. according to the formula 
net, = Sj Oj • Wjj , where wy Is the weight that connects 
neuron j to neuron i. Then the neuron computes its acti- 
vation Oj as a non-linear function of the total input 
according to the formula io 



where pi is a constant value, typical of neuron i. The 
above non-linear function is a sigmoid, which has an is 
expression of the type f(x) « lAI+e"""*®) and a value 
between 0 and 1. 

figure 2 shows a single neuron i with its forward 
connections, along which rt propagates the activation 
differences, and with its memory structures Ml j and M2j 20 
required for the speeding up method. Mlj contains the 
activation value at time t, 0j(t), as In conventional neural 
networks, and M2i the value at the preceding time t-1 , 
0i(t-1). The other neurons in the network also have sim- 
ilar memory structures, for example M1|^ and M2k for 2s 
neuron k. 

Figure 3 depicts the quantization of the set of output 
values (co-domain) of the sigmoid transfer function of 
the neuron, with the purpose of quantizing the activation 
levels of the neurons, thus making it possible to recog- 30 
nise the condition of activation similarity at times t and t- 
1. required in order no signal is propagated. Preferably, 
the neurons do not propagate any signal when the 
quantized values at times t and t-1 are identteal. The 
elementary operation to speed up execution of the net- ss 
work Is thus accomplished. 

We shall now describe in detail the method of 
speeding up the execution of tiie neural network. 

The method requires storing the activation value of 
a neuron at time t-1 and comparing It with the one com- 40 
puted at time t. If the activation values differ by less ttian 
a predetermined threshold (preferably, are the same), 
the neuron carries out no activity, othenvlse It propa- 
gates the activation difference, multiplied by the Inter- 
connection weights, to the neurons It Is connected to. 4S 
Since tine activation of a neuron is a continuous non-lin- 
ear function of its input, usually a sigmoid 

f(x) = 1/(1+e-'^*®'), 

so 

it would be necessary to quantize tiie co<lomain of such 
a function to a pre-determined number of values, in 
order the condition of activation identity can be actually 
recognised (Figure 3). Quantization is generally carried 
out for tiie neurons of tiie intermediate tevel(s), so that ss 
the speeding up concerns such l6vei(s). 

This quantization of ttie sigmoid co-domain Intro- 
duces a sort of quantization noise into the network, but 
since information is very distributed, tills error tends sta- 



tistically to be compensated and it does not bring about 
an appreciable degradation in performance. 

Ot)vk)usly, tiie number of quantization values must 
be estimated empirically: tiie smaller rt is. the more the 
mettiod accelerates; however, it cannot be excessively 
small to avoid a degradation in performanca In tiie case 
of realistic Multi-layer Perceptron networks, witti about 
50,000 - 100,000 weights, this number can vary from 
about 25 to about 50. 

We shall now describe In detail tiie execution of the 
neural network at generic time t, subdivided into the 
cycles on input neurons j, on intermediate neurons m 
and on output neurons k. 

A) Cycle on input neurons: 

1 . for all input neurons j the following steps are 
carried out: 

1.1 tiie activatton 0|(t) of the Input neuron, 
equal to the j*^ value of the signal quan- 
tized at time t is defined, after storing the 
previous activation value 0j(t-1): 

1.2 for each intermediate neuron m 
reached from neuron j, the total Input netf^ 
Is increased by the contribution coming 
from input neuron j, given by the difference 
Oj(t) - Oj(t-l) multiplied by interconnection 
weight w^^^. 

B) Cycle on Intermediate neurons: 

2. for ail intermediate neurons m. the following 
steps are performed: 

2.1 tiie quantized activation Is computed 
Witt! the formula 0^ » f "^^'(net^) . 
where f^*"*(x) is the sigmoid quantized on 
tiie co-domain, after storing the previous 
quantized activation Om*'"**"'(t-1): 

2.2 If o ^ '^^(t) = 0^ ^"^{t-D , nothing is 
done for neuron m. and tiie subsequent 
neuron Is considered, thus accomplishing 
the basic speeding-up operation: other- 
wise, the difference Is propagated; 

2.3 for each neuron k reached from neuron 
m, tiie total input netk Is increased by the 
contribution coming from neuron m, which 
contribution is equal to the difference 
Om'"'^(t)-Om^"'*^(t-l) multiplied by 
interconnection weight Wfan- 

C) Cycle on tiie output neurons: 

3. for all output neurons k. the exact activation 
is computed witii ttie formula OKsfCnet^) 
where f(x) is tiie sigmoid. 
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This method is repeated for all sampled values of 
the correlated input signal, from start time I =i 1 to end 
time t = T . where T constitutes the number of elements 
of the sampled input signal. Before starting to execute 
the network on a signal, values netj are initialised to q; s 
for each neuron In the networK as in the difference 
propagation mechanism the constant value q of each 2. 
neuron is defined at the start and It is never altered 
thereafter. 

The main advantages of the method are the follow- 10 

ing: 

a. optimisation is due to a conceptual modification 

of the execution of the network and not to a soft- 3. 
ware optimisation, so that the advantage is red and is 
it can be transposed on any hardware platform 
(even on DSP processor); 

b. the use of additional memory is of the order of the 
network units, i.e. of the neurons, and not of the 
order of the weights, so that it Is very limited; 20 

c. by appropriately choosing the quantization step 
of the co-domain of the sigmoid, it Is possible to 
obtain remarkable reductions in execution time 
without appreciable losses in performance; 

d. the method can be applied to all cases of zs 
sequential input with continuous dynamics and 
stow evolution (e.g. voice, video, sonar, medical sig- 
nals); 

e. In the worst case of complete and continuous 
variation in the input signal, the method converges 30 
to the limit case of a standard execution of the neu- 
ral networK without appreciable additional costs. 

It is evident that what has been described is pro- 
vided purely by way of non-limiting example. Variations 3S 
and modifications are possible without departing from 
the scope of the invention. 

Claims 

40 

1 . A method of speeding up the execution of a neural 
network for conrelated signal processing, wherein 
the network is made of neurons arranged according 
to a multilevel structure conprising, from a bottom 
level to a top level, an input level, one ore more 45 
intermediate levels and an output level, and the 
neurons in a level are connected to neurons in an 
immediately upper level by means of links of differ- 
ent intensities (synaptic weights), characterised in 
that the execution of the neural network at a generic so 
sampling instant t comprises, at least for neurons in 
the intermecfiate level or levels, the steps of: 

- computing and storing an activation value of 

the neuron at instant t; ss 4. 

- comparing the activation value at instant t with 
the activation value at a preceding sampling 
Instant t-1 and computing the difference 
between said values; 



propagating towards neurons in an immedi- 
ately upper layer said difference, weighted by 
the synaptic weights of the links with said neu- 
rons in the upper layer, only if said difference 
exceeds a given threshold. 

A method according to claim 1. characterised in 
that said activation values are quantized activation 
values and in that said comparison is carried out on 
the quantized values and the activation value differ- 
ence is propagated whenever the quantized values 
at instants t and t-1 are not klentical. 

A method according to claim 1 or 2. characterised 
In that the execution of the neural network at instant 
t comprises the following operations: 

a) for all input neurons j: 

a1) defining the activation value Oj(t) of the 
input neuron, equal to the j^^ value of the 
signal quantized at time t, after storing the 
previous activation value Oj(t-1); 
a2) increasing, for each intermediate level 
neuron m reached from neuron j. the total 
input netfn by the contribution coming from 
the input neuron j. said contribution being 
equal to the cfifference Oj(t) - o j(t-1) multi- 
plied by interconnection weight w^f, 

b) for all intermediate level neurons m: 

b1) computing a quantized activation value 
witii the formula o„ ^^"^ = f "^^""(net^) . 
where f^"^"\x) sigmoid ti-ansfer func- 
tion of the neuron being considered, quan- 
tized on the co-domain, after storing tiie 
previous quantized activation value 
Or„^"«"»(t.1); 

b2) if the quantized activation value 
o„, *^(t) = o„ ^'"""(t-l) , performing no 
operation for neuron m, and passing to the 
subsequent neuron: 

b3) increasing, for each neuron k reached 
from neuron m, the total input net^ by the 
contribution coming from neuron m, said 
contribution being equal to the difference 
o^'^^W-o^^^'^O-l) multiplied by 
interconnection weight w^^; 

c) for all output neurons k. computing the exact 
activation value with the formula o^ » f(net J 
where f(x) is the sigmoid. 

Method according to claim 3. characterised in that it 
is iterated for all sampled values of the con-elated 
input signal. 
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Method according to claim 3 or 4, characterised in 
that before starting network execution on a signal, 
the inputs are initialised by setting netj > Oj for 
each network neuron, Oi being a constant value for 
each neuron, which value is defined at the begin- 5 
ning and is not subsequently changed. 

Method according to any of claims 3 to 5, character- 
ised in that the co-domain of the sigmoid Is quan- 
tized with a number of quantization steps ranging 10 
from 25 to 50. 
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